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0 Method and means for accessing arrays of DASDS. 



@ In a method and means for accessing arrays of field HA, a record zero field RO, one or more count 

DASDs, write update of variable length records fields C and one or more data fields D. Each count 

stored in row major order on an array of N DASDs is field C is followed by a variable length block data 

facilitated by utilising the correlation between byte field D. Parity is recorded in the data field in phys- 

offsets of a variable length record and the byte offset ical track N in the DASD N. If data field D3 is to be 

of a byte level parity image of data stored on the updated and is five bytes long, the old and new 

same track across N-1 other DASDs. values of D3 are XORed and the old value of the last 

Each DASD physical track has a home address five bytes in 08 Is changed to match this. 

ARRAY WITH VARIABLE LENGTH RECORDS 
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This invention relates to methods and means 
for accessing arrays of direct access storage de- 
vices (DASDs). 

Tuning DASD Array Data Rate and Concurrency 
With Respect to Fixed Length Records and the 
Uki 

EP -A- 0458554, published before the filing 
date of this application, but after the priority date 
thereof, describes a method and means for acces- 
sing an array of N synchronous DASDs storing 
blocked data (data In the form of fixed equal length 
extents). The array appeared to the accessing CPU 
as a virtual DASD in which each track length was 
N*single DASD physical track length. The data rate 
was equal to (N-1)*single device rate. The model 
assumed that a single DASD (parity DASD or 
equivalent space) would facilitate data regeneration 
given a single DASD failure. A single parameter 
relating to the repetition pattern of storing the fixed 
blocks on the physical array can be used to "tune" 
both data rate and concurrency (number of in- 
dependent concurrent accesses). 

The CPU/array data rate Is maximised by ac- 
cessing N blocks at a time from N counterpart 
physical tracks of N DASDs, the accessed blocks 
forming part of a logical record of K*N consecutive 
fixed blocks with repetition intervals 1 to N, (N + 1) 

to 2N. (2N + 1) to 3N {(K-1)N + 1} to KN. Each 

physical track had a capacity of up to K blocks. 
This laying across of N blocks at a time Is termed 
"array column major order" with a repetition Inter- 
nal of N. Of course only one transaction at a time 
can be served as all N DASDs are being accessed 
by the same process. 

Concurrency is improved when consecutive 
blocks are laid out along a track of counterpart b < 
N DASDs, the repetition interval being K blocks. 
That is, the first track of the first DASD stores 
blocks 1 to K; a counterpart track of the second 
DASD stores blocks K+1 to 2K; and the coun- 
terpart track of the bth DASD store blocks (b-1)- 
K + 1 to bK. The laying out of blocks in the track 
direction is termed "array row major order". Con- 
sequently, the array can be partitioned into two 
groups of b and N-b DASDs respectively. Each 
group can be independently and concurrently ac- 
cessed. 

The solution of EP -A- 0458554 involves (a) 
formatting the blocks onto the array using a row 
major order modulus, and (b) executing the random 
sequences of large and small access requests over 
the array. More comprehensively stated, the meth- 
od Includes (a) formatting the K*N blocks of a 
logical file onto the array in row major order 
modulo M and in column major order modulo M*N, 
M lying in the closed Interval ( 1 ,K); and (b) execut- 
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ing large and small access requests over the N 
DASD array whereby the minimum number X of 
blocks transferred to achieve the maximum data 
rate for a given M lies In the closed interval ((N-1 , 

6 (N-1)K), the end points of the latter interval being 
defined by M = 1 and M = K respectively. 

As those skilled in the art readily appreciate, 
the method clearly equated high data rate, with 
numerically intensive computing (NIC). It further 

10 equated concurrency with transaction processing. 

In NIC, a CPU sequentially accesses and pro- 
cesses large messages formed from long strings of 
alphanumerics from fast access, high capacity, in- 
termediate result storage i.e. DASD arrays. Also, 

75 CPUs frequently use iterative rather than recursive 
algorithms. While both algorithnn types may ex- 
ecute repeated sequential referencing and re-re- 
ferencing of the same or similar strings of partial 
results, iterative based computations are less prone 

20 to data access errors propagating through the en- 
tire computation. Thus, access error might very 
well be ignored as a matter of system or applica- 
tion choice rather than interrupting or repeating a 
lengthy and involved CPU/array exchange. Another 

25 approach outside this discussion might be the use 
of block rather than cyclic error codes. Under this 
circumstance, the CPU would have to rebuild only 
the affected data block. 

Transactions are characterised by random stor- 

30 age referencing and atomic operations to ensure 
data integrity and recoverability. Such processing 
typically Involves an interactive man/machine ex- 
change of small/short messages occurring at an 
automatic bank teller machine, a supermarket 

35 checkout counter or the like. It is the sheer volume 
of such individually processed single thread com- 
putations that imposes a concurrency of access 
requirement on intermediate result storage (DASD 
array). That is, the need to process as many trans- 

40 actions as possible about the same time. 

A significant fraction of data reposes on DASD 
storage in variable length format. One regimen, 
which is used on IBM S/370 CPUs and attached 
externa! storage, is known as Count/Key/Data or 

45 CKD operating under the well known MVS operat- 
ing system. In this regimen, each record consists 
of fixed length count and key fields and a variable 
length data field. The count field defines the length 
of the data field while the key field serves as a 

50 record ID. The fields, as recorded on DASD, are 
separated on the track by a space or gap which 
defines a time interval during which the system 
prepares to handle the next field. 

A record having a data field spanning more 

55 than a physical DASD track is reformatted at the 
CPU into several smaller records with appropriate 
pointers or other linking conventions maintained 
somewhere in the storage or file management por- 



EP 0 503 768 A1 



tions of the operating system. Likewise, a track 
may store many small records of various lengths. 
This leads to the fact that reading, writing, and 
updating variable length records on files result in a 
more complex operation than that involving fixed 
blocks. 

In an article by Patterson et al, "A Case for 
Redundant Arrays of Inexpensive Disks (RAID)", 
ACM SIGMOD Conference, Chicago Illinois, June 
1- 3, 1988 logical record to physical track mapping 
is discussed and there is mentioned storing blocks 
onto a DASD array in column major order avoiding 
both mirrored DASDs and Hamming ECC encod- 
ing. This was in order have a data rate accom- 
modating both small and large access requests. 
Also described is the calculation of new parity as 
the XORing of old data, new data, and old parity. 
The latter is significant because it Indicates that It 
Is not necessary to access N DASDs if all that Is 
required is the updating of one block on one 
DASD. As pointed out in EP -A- 0458554, only the 
affected data and parity DASD containing the in- 
formation of interest need be accessed. 

Clearly, two partitions of b and N-b DASDs can 
be concurrently accessed by respective applica- 
tions. However, because the arms of the DASDs 
within a partition may be differently positioned on 
the data and parity tracks, there Is no concurrency, 
or more properly, synchronism of the elements In 
positioning, reading, or writing. 

The term "single pass" as used in this speci- 
fication means an interval during which all the 
operations necessary to effectuate a given end 
effect or result are performed. The term does not 
connote a single DASD track revolution. 

Accordingly this invention seeks to devise a 
method and means for for ensuring single pass 
access to a selected DASD in an array of N DASDs 
especially suited to small track read and write 
operations and where data is recorded in variable 
length format. 

The foregoing object is satisfied by a method 
and means utilising the correlation between byte 
offsets of a variable length record and the byte 
offset of a byte level parity image of data stored on 
the same track across N-1 other DASDs. 

The method of the invention write updates se- 
lected fields among variable length formatted 
(count, key, and data) records stored in row major 
order on DASD tracks of an array of N DASDs. The 
method and means comprising the steps/elements 
of generating a byte level exclusive OR simple 
parity image of the records as physically recorded 
from each Ith track across N-1 DASDs and storing 
said image' as a variable length record on an ith 
track of the Nth DASD. 

The method further comprises the step, re- 
sponsive to each write request to a selected DASD, 



of (1) establishing byte position offset synchrony of 
the record to be updated on the ith track of the 
selected DASD and the record identity within the 
parity record on the ith track of the Nth DASD, (2) 
5 obtaining and updating the records from the se- 
lected DASD and from the Nth DASD; and (3) 
rewriting the updated data and parity records back 
to ith track on the selected and Nth DASDs, re- 
spe"ctively. 

10 Restated, the write update in a single pass is 

obtained by altering and rewriting the parity con- 
current with altering and rewriting the data. That Is, 
both data and relevant parity are accessed in terms 
of byte offsets in an equivalent virtual DASD. Then, 

75 the data and parity are recalculated and rewritten 
on the selected and Nth DASD in synchronized 
fashion, respectively. 

The parity images are distributed across dif- 
ferent DASDs such that there Is no "parity DASD" 

20 as such. For instance, for an array of N = 10 
DASDs, the image of the ith track from DASDs 1 to 
9 would be stored on DASD 10 while the Image of 
the track over DASDs 2 to 10 would be (i + 1)th 
stored on DASD 1. 

25 Also, full track read of parity image may be 
avoided where the Image is stored over two instead 
of a single image track. 

Figure 1 depicts diagrammatically a synchro- 
nous array of N DASDs according to the prior 

30 art; 

Figure 2 illustrates a prior art DASD array at- 
tached to a CPU using a cache and a controller 
in the data path; 

Figures 3 and 4 illustrate show prior art fixed 
35 block layout in respective DASD array column 
and row major order; 

Figure 5 illustrates a variable length record lay- 
out across the DASD array using full track 
record on parity track according to the invention; 
40 and 

Figure 6 illustrates a variable record layout using 
two parity tracks according to the invention. 



45 



Logical and Physical Aspects of A DASD Array 



In a synchronous array of N DASDs according 
to the prior art (Figure 1), the blocks of recorded 
data are written in column major order with the Nth 
DASD always containing the parity block, as in EP 

50 -A-0458554. Advantageously, the data rate is in- 
creased N-1 times the transfer rate of a single 
DASD and has found use where large read/writes 
were involved. 

A CPU 1 (Figure 2) accesses DASDs 1 to N 

55 over a path including channel 3, array controller 5 
and cache 13. Controller 5 operatlvely secures 
synchronism and accesses among DASDs 1 to N 
over access and control path 7. Responsive to an 



EP 0 503 768 A1 



access, N bytes of data can be exchanged in 
parallel to cache 13 over data path 15. Likewise, 
data can be exchanged serially by byte between 
CPU 1 and cache 13 over control and data paths 9 
and 1 1 via the controller 5 and path 3. 

Placement of cache 13 in the array alters the 
view of storage as seen by CPU 1, The cache 13 
smooths and facilitates any application running on 
CPU 1 accessing information in a block organised 
logical DASD having one or more logical tracks 
organised Into logical cylinders. The smoothing de- 
rives from the cache operating as a specialised 
buffer decoupling the cache/array interaction from 
the CPU/cache interface. That is, it reduces the 
data rate differences as cache reference, rate to the 
DASD array should be significantly less than the 
CPU reference rate to the cache for at least ran- 
dom (non-sequential) accesses. 

KN blocks (Figure 3) are formatted in column 
major order. Each parity block spans the N-1 other 
blocks in its column. However, the K parity blocks 
are diagonally striped across the DASD. In the 
event of concurrent write operations, the diagonal 
striping avoids making any one DASD contentious, 
as would be the case if all parity blocks were on a 
parity DASD. 

Where K » N, then the striping would occur K 
modulo N. Furthermore as mentioned above, cur- 
rent read and write of different blocks in the same 
column (as they are located on different DASDs) is 
possible. Notwithstanding, concurrent writes to 
blocks in the same column or to blocks on the 
same DASD are to be avoided. In the first case, the 
same parity block can only be changed serially, 
while In the second case the same DASD can only 
be written or read in serial order. 

The mapping of KN blocks (Figure 4), of a 
logical track onto an array of N DASDs of the type 
2H in row major order K modulo M where M = K 
shows a row of K parity blocks on the Nth DASD 
spanning counterpart column blocks. The array has 
N DASDs, K blocks per physical track, and NK 
blocks per logical track. However, unlike the col- 
umn major order layout order of type 2V array, the 
KN consecutive blocks of the logical track are 
stored in row major order along a counterpart phys- 
ical DASD track. Also, the parity blocks in the 2H 
array are different from those in the 2V array. For 
the KN blocks of the group or logical track shown 
in the K parity blocks are stored on the physical 
track of DASD N. Then, for the KN blocks of a 
second group or logical track (not shown), the K 
parity blocks would be stored on a predetermined 
track on DASD N-1. Similarly, the KN blocks of a 
third group or logical track would have their K 
parity blocks stored on DASD N-2 and so on. 

Variable Block Fonnatting in Row Major Order and 



Write Updating Using A Single Parity Track 

With fixed block sizes, It becomes easy to 
create and store parity blocks. When records on 

5 the different physical tracks are ail of different 
sizes, it is not clear how the data on the parity 
track is to be stored, as the records on the different 
tracks do not all line up. The method of the present 
invention facilitates the storage and access of vari- 

10 able length blocks in the DASD DASD array. 

The layout of data records (Figure 5) stored in 
a variable length format CKD, more closely resem- 
bles the row major rather than the column major 
order. That is, consecutive records are stored along 

75 a physical track. Unlike the constant block size 
case however, the number of records in a physical 
track Is a variable number and is not fixed at K. 

In formatting DASD disk tracks using the CKD 
convention, each physical track Includes Home Ad- 

20 dress (HA) and Record Zero (RO) fields, followed 
by some number of CKD records. For the logical 
track shown, all the parity is stored on the physical 
track of DASD N. For the next logical track, all the 
parity is stored in DASD N-1, and so on, Signifi- 

25 cantly, parity is spread among all the DASDs, and 
there is no one single parity DASD. This scheme is 
very similar to that for row track layout. 

Any description of the data stored in the parity 
track on the Nth DASD should cover the case 

30 where individual physical tracks may have defects 
In different locations. However, the layout shown 
assumes that there is no defect on any of the 
physical tracks making up the logical track under 
consideration. 

35 The first two fields on the parity track are HA 
and RO, respectively, contain Information regarding 
the physical address of the track and the position 
of defects in it, and are generated in the normal 
manner (and not as the parity of HA and RO fields 

40 from the other N-1 tracks). Following the HA and 
RO, there is a single full-track record consisting of a 
count field (C8) and a data field (D8). 

A count field on a CKD track contains physical 
information (physical address, defect pointers, etc.) 

45 and logical information (five bytes in the format 
CCHHR, that is, two bytes of cylinder number, two 
bytes of head number and one byte of record 
number). 

The physical information part of the count field 
50 (C8) of the parity track contains the obvious values. 
The five bytes of logical infomnation in the count 
field of the parity track (CCHHR) is defined as 
equal to the parity PCOR) of the five bytes of 
CCHHR from the count fields of the first record 
55 after RO from all the other N-1 tracks. This com- 
pletely defines the contents of C8. Next, the con- 
tents of the data field (D8) of the parity track are 
defined. 
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In this specification, the notation "< = " nneans 
"less than or equal to". That is, it defines a closed 
or bounded number interval. Also, the acronym 
ECC, stands for any type of "error correction 
code" ordinarily and usually appended to a record. 
Among error correction codes of the block type are 
Hamming codes and BCH codes. 

For 1 < = i < = N, let B(i) represent the byte 
position corresponding to the" location where the 
first data field after RO begins on track I (DASD i). 
All the B(t) byte positions would be identrcal if there 
is no defect In any of the N physical tracks. How- 
ever, the presence of defects will make 8(1) vary 
with i. 

Let E(i) be the byte position corresponding to 
where the last byte of data on a particular physical 
track may be stored, 1< = i < = N. If there is no 
defect on a track, E(i) is several bytes from the end 
of the physical track. If a track has its full com- 
plement of defects, then E(i) is ECC bytes before 
the last byte on the track, where ECC is the num- 
ber of bytes needed to store the ECC on the data 
field. 

It is important that, for 1 <= i <= N, the 
number of bytes between B(i) and E(i), not includ- 
ing defects, is the same. If" X is th*e number of 
bytes, then D8 is X bytes long. 

Byte B(i) on the parity track is the XOR of byte 
B(i) from alfthe other N-1 tracks. Also, byte B(l) + 1 
is "the XOR of byte B(i) + 1 from all other "N-1 
tracks. In this way. X bytes in the data field of the 
parity track can be generated, by doing a byte by 
byte XOR of every non-defective byte from B(i) to 
E(i), for 1 < = i < = N. A byte position that fall's in 
the gap between fields on any of the physical 
tracks or a byte position that is part of an ECC field 
is assumed to be a byte of zeros for the purpose of 
generating parity. 

If it Is desired to update D3 (Rgure 5) on 
DASD 1 and D3 be five bytes long, D3 is read from 
DASD 1, and D8 is read from DASD N. Field D8 is 
updated by changing the value of the last five 
bytes in D8. Significantly, this Is determined 
through the XORing of the old and new values of 
D3 and the old value of the last five bytes of D8. 
This updated value of D8, and the new value of D3 
are then written back to DASD. 

Data Reconstruction 

When a physical track on a failed DASD disk 
needs to be recreated, the corresponding physical 
track from the surviving N-1 DASDs is copied Into 
N-1 full-track buffers. Each physical track of data 
that is read is stored in its corresponding buffer, 
with every field on the track separated by gaps as 
they would be on the physical track. Zeros are 
stored in the buffer where gaps exist between 



fields (ECC is considered as part of the gap). It Is 
Important to store each of the N-1 physical tracks 
in the buffers with gaps between fields (rather than 
with all the fields adjacent to each other), because 
6 this causes all the data in the N-1 buffers to be 
aligned properly for a byte-by-byte recreation of 
the missing track to be generated. 
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Write Updating Using Two Parity Tracks 



One limitation of a single track byte level sim- 
ple parity image for each group of N-1 counterpart 
data tracks as used with variable-length records is 
that small writes will force full-track reads and 

T5 writes of the parity track. In cpntrast, fixed block 
systems avoid such a requirement because only 
the relevant parity block need be read rather than 
the entire parity track. 

In a variation of the method of this Invention 

20 (Figure 6), all the information (X bytes) stored in 
the large data field D8 is partitioned and stored as 
records of equal size (say 4 kbytes). For example, 
if X. the size of D8. was 12 kbytes, then store 
those 12 kbytes In three 4 kbyte records. If the 

25 information stored in D8 is stored as three 4 
kilobyte records instead of as a single, large 
record, it is clear that all the information in D8 will 
not fit into a single parity track (because of the 
gaps and count fields between the 4 kilobyte 

30 records). Instead, two parity tracks would be used 
to store all the X bytes contained in D8 in 4 
kilobyte records. Fields D8, D9 and D10 (Figure 6) 
contain the same bytes that were stored in field D8 
(Figure 5). 

35 With this variation using two parity tracks per 
logical track, controller 5 would have to ascertain 
which one of the multiple 4 kilobyte record fields 
from the parity tracks need to be read and written 
when executing small writes. Then, it would access 

40 only these relevant 4 kilobyte records from the 
parity tracks, and full-track reads and writes of the 
parity track are no longer required. Thus, updating 
D3 would only require that data fields D3 and D10 
be read and written back. 

45 The invention is of particular application where 
data is stored in variable length format (CKD) and 
where many of the transfers are small or bursts. 
Small transfers (read/write operations) occur when 
less than all the array DASDs in an otherwise 

60 synchronous transfer group are accessed. 

ClainDS 

1. A method for write updating records among 
55 variable length formatted (count, key, and data) 

records stored in row major order on DASD 
tracks of an array of N DASDs, comprising the 
steps of: 
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(a) generating a byte level exclusive OR 
simple parity image of the variable length 
records from each Ith track across N-1 
DASDs and storing the image on an ith 
track of the Nth DASD; and ' 5 

(b) responsive to each write request to a 
selected DASD, 

(1) establishing byte position offset syn- 
chrony of the record to be updated on 

the ith track of the selected DASD and io 
the record identity within the parity 
record on the ith track of the Nth DASD, 

(2) obtaining "and updating the records 
from the selected DASD and from the 

Nth DASD; and 75 

(3) rewriting the updated data and parity 
records back to ith track on the selected 
and Nth DASDs. respectively. 

2. A method according to claim 1, wherein step 20 
(a) includes the steps of generating r Images In 
one-to-one relation to r groups of variable 
length records stored on counterpart tracks, 
each group Including an image stored on an ith 
track of a first DASD and variable length 25 
records on the ith tracks of (N-1) other coun- 
terpart DASDs; "and storing each generated 
image such that: 

(1) for r > N, each DASD stores at least one 
image "track and at least one DASD stores 30 
two image tracks; and 

(2) for r < = N. no two images formed from 
two dissimilar groups of variable length 
records are stored on the same DASD, each 
group being stored on respective ith and jth 35 
tracks of combinatorially distinctive groups 

of N-1 DASDs. 

3. A method according to claim 1 or 2, wherein 
step (a) includes the steps of blocking and 40 
storing an image over two physical tracks on 

the Nth DASD, so as to minimise the neces- 
sary reading of a full track. 

4. A method according to claim 1, 2 or 3, includ- 45 
ing further the step of (c) recreating the vari- 
able length records stored on the ith physical 
track of any failed DASD in the array by: 

(1) copying the variable length records from 
each of the surviving N-1 DASDs into N-1 50 
full track buffers in which the relative posi- 
tion of each gap separated field Is pre- 
served by a predetermined Boolean value 
string inserted therebetween, thus retaining 
field alignment among tracks; and 55 

(2) forming an image of the records on the 
unavailable track and their track position 
relationship by XORing the contents of the 



N-1 full track buffers. 

5. A method according to claim 1, 2, 3 or 4, 
wherein the update of the parity includes the 
XORing of the record being accessed, any 
change to the record being accessed, and the 
parity being accessed. 

6. A method for ensuring single pass access to a 
selected one of an array of N DASDs espe- 
cially suited for small track read and write 
operations, the array Including N-1 DASDs 
having data recorded thereon as variable 
length format (CKD) records with row track 
layout, and an Nth DASD containing parity of 
the data recorded upon the N-1. other DASDs, 
comprising the steps of: 

(a) creating a virtual image as a byte-or- 
ganised result of an exclusive OR operation 
upon counterpart bytes on the counterpart 
tracks across N-1 DASDs. and mapping the 
virtual image into at least one variable 
length formatted record onto the Nth DASD; 
and 

(b) responsive to a write operation acces- 
sing a record on the Ith track on a selected 
one of the N-1 DASDs, 

(1) ascertaining the byte offset of the 
record to be updated on the ith track of 
the selected DASD and the record iden- 
tity and byte offsets within the parity 
record on the ith track of the Nth DASD. 
obtaining the record to be updated from 
the selected DASD and the parity 
records from the Nth DASD, and execut- 
ing updates against the obtained records; 
and 

(2) rewriting the updated parity and data 
records back to ith track on the Nth and 
selected DASD respectively. 

7. A method according to claim 6, wherein step 
(a) includes the step of writing each of r virtual 
images in one-to- one relation onto" tracks 
across r <= N DASDs, where each of the r 
images ~is the byte XORed result over records 
stored on the Ith counterpart tracks on N-1 
other DASDs, w'here r <= N no DASD stores 
more than one image, or where r > N each 
DASD stores at least one image and at least 
one DASD stores two Images. 

8. A system comprising an array of N DASDs, a 
CPU, and means responsive to requests from 
the CPU for accessing variable length records 
stored in row track order on counterpart DASD 
tracks in the array, characterised by 
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means for establishing a byte position cor- 
relation between variable length records stored 
on the ith track of each of (N-1) DASDs and a 
simple 'byte level parity image thereof stored 
as a variable length record on the ith track of s 
the Nth DASD; and 

means responsive to a write request from 
the CPU through the accessing means for ac- 
cessing a record on the ith track on a selected to 
one of the N-1 DASDs by 

(1) ascertaining the byte offset of the record 
to be updated on the ith track of the se- 
lected DASD and the record Identity and 
byte offsets within the parity record on the 75 
Ith track of the Nth DASD. obtaining the 
record to be updated from the selected 
DASD and the parity records frohn the Nth 
DASD, and executing updates against the 
obtained records; and 20 

(2) rewriting the updated data and parity 
records back to ith track on the selected 
and Nth DASDs respectively. 

25 
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